How general is the natural frequency effect? The case of joint probabilities.

Natural frequencies are known to improve performance in Bayesian reasoning. However, their impact in situations with two binary events has not yet been completely examined, as most researchers in the last 30 years focused only on conditional probabilities. Nevertheless, situations with two binary events consist of 16 elementary probabilities and so we widen the scope and focus on joint probabilities. In this article, we theoretically elaborate on the importance of joint probabilities, for example, in situations like the Linda problem. Furthermore, we implemented a study in a 2×5×2 design with the factors information format (probabilities vs. natural frequencies), visualization type ("Bayesian text" vs. tree diagram vs. double tree diagram vs. net diagram vs. 2×2 table), and context (mammography vs. economics problem). Additionally, all four "joint questions" (i.e., P(A∩B),P(A¯âˆ©B),P(A¯âˆ©B¯),P(A∩B¯)) were asked for. The main factor of interest was whether there is a format effect in the five visualization types named above. Surprisingly, the advantage of natural frequencies was not found for joint probabilities and, most strikingly, the format interacted with the visualization type. Specifically, while people's understanding of joint probabilities in a double tree seems to be worse than the understanding of the corresponding natural frequencies (and, thus, the frequency effect holds true), the opposite seems to be true in the 2 × 2 table. Hence, the advantage of natural frequencies compared to probabilities in typical Bayesian tasks cannot be found in the same way when joint probability or frequency tasks are asked.

Measuring people's covariational reasoning in Bayesian situations.

Previous research on Bayesian reasoning has typically investigated people's ability to assess a posterior probability (i.e., a positive predictive value) based on prior knowledge (i.e., base rate, true-positive rate, and false-positive rate). In this article, we systematically examine the extent to which people understand the effects of changes in the three input probabilities on the positive predictive value, that is, covariational reasoning. In this regard, two different operationalizations for measuring covariational reasoning (i.e., by single-choice vs. slider format) are investigated in an empirical study with N = 229 university students. In addition, we aim to answer the question wheter a skill in "conventional" Bayesian reasoning is a prerequisite for covariational reasoning.

Bayesian versus diagnostic information in physician-patient communication: Effects of direction of statistical information and presentation of visualization.

BACKGROUND: Communicating well with patients is a competence central to everyday clinical practice, and communicating statistical information, especially in Bayesian reasoning tasks, can be challenging. In Bayesian reasoning tasks, information can be communicated in two different ways (which we call directions of information): The direction of Bayesian information (e.g., proportion of people tested positive among those with the disease) and the direction of diagnostic information (e.g., the proportion of people having the disease among those tested positive). The purpose of this study was to analyze the impact of both the direction of the information presented and whether a visualization (frequency net) is presented with it on patient's ability to quantify a positive predictive value. MATERIAL AND METHODS: 109 participants completed four different medical cases (2⨯2⨯4 design) that were presented in a video; a physician communicated frequencies using different directions of information (Bayesian information vs. diagnostic information). In half of the cases for each direction, participants were given a frequency net. After watching the video, participants stated a positive predictive value. Accuracy and speed of response were analyzed. RESULTS: Communicating with Bayesian information led to participant performance of only 10% (without frequency net) and 37% (with frequency net) accuracy. The tasks communicated with diagnostic information but without a frequency net were correctly solved by 72% of participants, but accuracy rate decreased to 61% when participants were given a frequency net. Participants with correct responses in the Bayesian information version without visualization took longest to complete the tasks (median of 106 seconds; median of 13.5, 14.0, and 14.5 seconds in other versions). DISCUSSION: Communicating with diagnostic information rather than Bayesian information helps patients to understand specific information better and more quickly. Patients' understanding of the relevance of test results is strongly dependent on the way the information is presented.

Improving Diagnostic Efficiency with Frequency Double-Trees and Frequency Nets in Bayesian Reasoning.

Background. Medical students often have problems with Bayesian reasoning situations. Representing statistical information as natural frequencies (instead of probabilities) and visualizing them (e.g., with double-trees or net diagrams) leads to higher accuracy in solving these tasks. However, double-trees and net diagrams (which already contain the correct solution of the task, so that the solution could be read of the diagrams) have not yet been studied in medical education. This study examined the influence of information format (probabilities v. frequencies) and visualization (double-tree v. net diagram) on the accuracy and speed of Bayesian judgments. Methods. A total of 142 medical students at different university medical schools (Munich, Kiel, Goettingen, Erlangen, Nuremberg, Berlin, Regensburg) in Germany predicted posterior probabilities in 4 different medical Bayesian reasoning tasks, resulting in a 3-factorial 2 × 2 × 4 design. The diagnostic efficiency for the different versions was represented as the median time divided by the percentage of correct inferences. Results. Frequency visualizations led to a significantly higher accuracy and faster judgments than did probability visualizations. Participants solved 80% of the tasks correctly in the frequency double-tree and the frequency net diagram. Visualizations with probabilities also led to relatively high performance rates: 73% in the probability double-tree and 70% in the probability net diagram. The median time for a correct inference was fastest with the frequency double tree (2:08 min) followed by the frequency net diagram and the probability double-tree (both 2:26 min) and probability net diagram (2:33 min). The type of visualization did not result in a significant difference. Discussion. Frequency double-trees and frequency net diagrams help answer Bayesian tasks more accurately and also more quickly than the respective probability visualizations. Surprisingly, the effect of information format (probabilities v. frequencies) on performance was higher in previous studies: medical students seem also quite capable of identifying the correct solution to the Bayesian task, among other probabilities in the probability visualizations. Highlights: Frequency double-trees and frequency nets help answer Bayesian tasks not only more accurately but also more quickly than the respective probability visualizations.In double-trees and net diagrams, the effect of the information format (probabilities v. natural frequencies) on performance is remarkably lower in this high-performing sample than that shown in previous studies.

Tversky and Kahneman's Cognitive Illusions: Who Can Solve Them, and Why?

In the present paper we empirically investigate the psychometric properties of some of the most famous statistical and logical cognitive illusions from the "heuristics and biases" research program by Daniel Kahneman and Amos Tversky, who nearly 50 years ago introduced fascinating brain teasers such as the famous Linda problem, the Wason card selection task, and so-called Bayesian reasoning problems (e.g., the mammography task). In the meantime, a great number of articles has been published that empirically examine single cognitive illusions, theoretically explaining people's faulty thinking, or proposing and experimentally implementing measures to foster insight and to make these problems accessible to the human mind. Yet these problems have thus far usually been empirically analyzed on an individual-item level only (e.g., by experimentally comparing participants' performance on various versions of one of these problems). In this paper, by contrast, we examine these illusions as a group and look at the ability to solve them as a psychological construct. Based on an sample of N = 2,643 Luxembourgian school students of age 16-18 we investigate the internal psychometric structure of these illusions (i.e., Are they substantially correlated? Do they form a reflexive or a formative construct?), their connection to related constructs (e.g., Are they distinguishable from intelligence or mathematical competence in a confirmatory factor analysis?), and the question of which of a person's abilities can predict the correct solution of these brain teasers (by means of a regression analysis).

Natural frequency trees improve diagnostic efficiency in Bayesian reasoning.

When physicians are asked to determine the positive predictive value from the a priori probability of a disease and the sensitivity and false positive rate of a medical test (Bayesian reasoning), it often comes to misjudgments with serious consequences. In daily clinical practice, however, it is not only important that doctors receive a tool with which they can correctly judge-the speed of these judgments is also a crucial factor. In this study, we analyzed accuracy and efficiency in medical Bayesian inferences. In an empirical study we varied information format (probabilities vs. natural frequencies) and visualization (text only vs. tree only) for four contexts. 111 medical students participated in this study by working on four Bayesian tasks with common medical problems. The correctness of their answers was coded and the time spent on task was recorded. The median time for a correct Bayesian inference is fastest in the version with a frequency tree (2:55 min) compared to the version with a probability tree (5:47 min) or to the text only versions based on natural frequencies (4:13 min) or probabilities (9:59 min).The score diagnostic efficiency (calculated by: median time divided by percentage of correct inferences) is best in the version with a frequency tree (4:53 min). Frequency trees allow more accurate and faster judgments. Improving correctness and efficiency in Bayesian tasks might help to decrease overdiagnosis in daily clinical practice, which on the one hand cause cost and on the other hand might endanger patients' safety.

A New Visualization for Probabilistic Situations Containing Two Binary Events: The Frequency Net.

In teaching statistics in secondary schools and at university, two visualizations are primarily used when situations with two dichotomous characteristics are represented: 2 × 2 tables and tree diagrams. Both visualizations can be depicted either with probabilities or with frequencies. Visualizations with frequencies have been shown to help students significantly more in Bayesian reasoning problems than probability visualizations do. Because tree diagrams or double-trees (which are largely unknown in school) are node-branch structures, these two visualizations (in contrast to the 2 × 2 table) can even simultaneously display probabilities on branches and frequencies inside the nodes. This is a teaching advantage as it allows the frequency concept to be used to better understand probabilities. However, 2 × 2 tables and (double-)trees have a decisive disadvantage: While joint probabilities [e.g., P(A∩B)] are represented in 2 × 2 tables but no conditional probabilities [e.g., P(A|B)], it is exactly the other way around with (double-)trees. Therefore, a visualization that is equally suitable for the representation of joint probabilities and conditional probabilities is desirable. In this article, we present a new visualization-the frequency net-in which all absolute frequencies and all types of probabilities can be depicted. In addition to a detailed theoretical analysis of the frequency net, we report the results of a study with 249 university students that shows that "net diagrams" can improve reasoning without previous instruction to a similar extent as 2 × 2 tables and double-trees. Regarding questions about conditional probabilities, frequency visualizations (2 × 2 table, double-tree, or net diagram with absolute frequencies) are consistently superior to probability visualizations, and the frequency net performs as well as the frequency double-tree. Only the 2 × 2 table with frequencies-the one visualization that participants were already familiar with-led to higher performance rates. If, on the other hand, a question about a joint probability had to be answered, all implemented visualizations clearly supported participants' performance, but no uniform format effect becomes visible. Here, participants reached the highest performance in the versions with probability 2 × 2 tables and probability net diagrams. Furthermore, after conducting a detailed error analysis, we report interesting error shifts between the two information formats and the different visualizations and give recommendations for teaching probability.

An Eye-Tracking Study of Statistical Reasoning With Tree Diagrams and 2 × 2 Tables.

Changing the information format from probabilities into frequencies as well as employing appropriate visualizations such as tree diagrams or 2 × 2 tables are important tools that can facilitate people's statistical reasoning. Previous studies have shown that despite their widespread use in statistical textbooks, both of those visualization types are only of restricted help when they are provided with probabilities, but that they can foster insight when presented with frequencies instead. In the present study, we attempt to replicate this effect and also examine, by the method of eye tracking, why probabilistic 2 × 2 tables and tree diagrams do not facilitate reasoning with regard to Bayesian inferences (i.e., determining what errors occur and whether they can be explained by scan paths), and why the same visualizations are of great help to an individual when they are combined with frequencies. All ten inferences of N = 24 participants were based solely on tree diagrams or 2 × 2 tables that presented either the famous "mammography context" or an "economics context" (without additional textual wording). We first asked participants for marginal, conjoint, and (non-inverted) conditional probabilities (or frequencies), followed by related Bayesian tasks. While solution rates were higher for natural frequency questions as compared to probability versions, eye-tracking analyses indeed yielded noticeable differences regarding eye movements between correct and incorrect solutions. For instance, heat maps (aggregated scan paths) of distinct results differed remarkably, thereby making correct and faulty strategies visible in the line of theoretical classifications. Moreover, the inherent structure of 2 × 2 tables seems to help participants avoid certain Bayesian mistakes (e.g., "Fisherian" error) while tree diagrams seem to help steer them away from others (e.g., "joint occurrence"). We will discuss resulting educational consequences at the end of the paper.

Why Can Only 24% Solve Bayesian Reasoning Problems in Natural Frequencies: Frequency Phobia in Spite of Probability Blindness.

For more than 20 years, research has proven the beneficial effect of natural frequencies when it comes to solving Bayesian reasoning tasks (Gigerenzer and Hoffrage, 1995). In a recent meta-analysis, McDowell and Jacobs (2017) showed that presenting a task in natural frequency format increases performance rates to 24% compared to only 4% when the same task is presented in probability format. Nevertheless, on average three quarters of participants in their meta-analysis failed to obtain the correct solution for such a task in frequency format. In this paper, we present an empirical study on what participants typically do wrong when confronted with natural frequencies. We found that many of them did not actually use natural frequencies for their calculations, but translated them back into complicated probabilities instead. This switch from the intuitive presentation format to a less intuitive calculation format will be discussed within the framework of psychological theories (e.g., the Einstellung effect).

A Balanced Risk-Benefit Analysis to Determine Human Risks Associated with Pyrrolizidine Alkaloids (PA)-The Case of Herbal Medicinal Products Containing St. John's Wort Extracts (SJW).

OBJECTIVES: Pyrrolizidine alkaloids (PA) exist ubiquitously in our environment. More than 6000 plants, about 3% of the world’s flowering plants, are known to synthesize PA. As a consequence, many herbal ingredients, including St. John’s wort (SJW), are contaminated with PA that can possess acute and subchronic toxic effects as well as mutagenic and genotoxic properties. Therefore, the possible benefits of SJW as an herbal remedy against depression need to be weighed against the possible risks of unwanted PA intake. METHODS: We searched the literature regarding the current knowledge on PA and evaluated the evidence on the antidepressant effects of quantified SJW extract based on a Cochrane Review and the current practice guidelines on depression. Risks are depicted in form of a risk ladder and benefits in form of an icon array. RESULTS: Evidence from clinical studies indicates that quantified SJW extract is an effective treatment option for mild to moderate depression with fewer side effects than conventional antidepressants. Health statistics from different countries do not quantify cases of death caused by PA intake. However, deaths due to suicide, often triggered by depression, are common (11 in 1000 in Germany in 2015) and rank between fatalities due to liver diseases (16 in 1000) and household accidents (10 in 1000). CONCLUSIONS: Quantified SJW extract is a safe and effective treatment option, and its potential of treating depression outweighs the (hypothetical) risk of unwanted PA intake.

Visualizing the Bayesian 2-test case: The effect of tree diagrams on medical decision making.

In medicine, diagnoses based on medical test results are probabilistic by nature. Unfortunately, cognitive illusions regarding the statistical meaning of test results are well documented among patients, medical students, and even physicians. There are two effective strategies that can foster insight into what is known as Bayesian reasoning situations: (1) translating the statistical information on the prevalence of a disease and the sensitivity and the false-alarm rate of a specific test for that disease from probabilities into natural frequencies, and (2) illustrating the statistical information with tree diagrams, for instance, or with other pictorial representation. So far, such strategies have only been empirically tested in combination for "1-test cases", where one binary hypothesis ("disease" vs. "no disease") has to be diagnosed based on one binary test result ("positive" vs. "negative"). However, in reality, often more than one medical test is conducted to derive a diagnosis. In two studies, we examined a total of 388 medical students from the University of Regensburg (Germany) with medical "2-test scenarios". Each student had to work on two problems: diagnosing breast cancer with mammography and sonography test results, and diagnosing HIV infection with the ELISA and Western Blot tests. In Study 1 (N = 190 participants), we systematically varied the presentation of statistical information ("only textual information" vs. "only tree diagram" vs. "text and tree diagram in combination"), whereas in Study 2 (N = 198 participants), we varied the kinds of tree diagrams ("complete tree" vs. "highlighted tree" vs. "pruned tree"). All versions were implemented in probability format (including probability trees) and in natural frequency format (including frequency trees). We found that natural frequency trees, especially when the question-related branches were highlighted, improved performance, but that none of the corresponding probabilistic visualizations did.

A Balanced Risk-Benefit Analysis to Determine Human Risks Associated with Pyrrolizidine Alkaloids (PA)-The Case of Tea and Herbal Infusions.

Humans are exposed to pyrrolizidine alkaloids (PA) through different sources, mainly from contaminated foodstuff. Teas and herbal infusions (T&HI) can be contaminated by PA producing weed. PA can possess toxic, mutagenic, genotoxic, and carcinogenic properties. Thus, possible health risks for the general population are under debate. There is a strong safety record for T&HI and additionally epidemiological evidence for the preventive effects of regular tea consumption on cardiovascular events and certain types of cancer. There is no epidemiological evidence, however, for human risks of regular low dose PA exposure. Recommended regulatory PA-threshold values are based on experimental data only, accepting big uncertainties. If a general risk exists through PA contaminated T&HI, it must be small compared to other frequently accepted risks of daily living and the proven health effects of T&HI. Decision making should be based on a balanced riskbenefit analysis. Based on analyses of the scientific data currently available, it is concluded that the benefits of drinking T&HI clearly outweigh the negligible health risk of possible PA contamination. At the same time, manufacturers must continue their efforts to secure good product quality and to be transparent on their measures of quality control and risk communication.

Effects of visualizing statistical information - an empirical study on tree diagrams and 2 × 2 tables.

In their research articles, scholars often use 2 × 2 tables or tree diagrams including natural frequencies in order to illustrate Bayesian reasoning situations to their peers. Interestingly, the effect of these visualizations on participants' performance has not been tested empirically so far (apart from explicit training studies). In the present article, we report on an empirical study (3 × 2 × 2 design) in which we systematically vary visualization (no visualization vs. 2 × 2 table vs. tree diagram) and information format (probabilities vs. natural frequencies) for two contexts (medical vs. economical context; not a factor of interest). Each of N = 259 participants (students of age 16-18) had to solve two typical Bayesian reasoning tasks ("mammography problem" and "economics problem"). The hypothesis is that 2 × 2 tables and tree diagrams - especially when natural frequencies are included - can foster insight into the notoriously difficult structure of Bayesian reasoning situations. In contrast to many other visualizations (e.g., icon arrays, Euler diagrams), 2 × 2 tables and tree diagrams have the advantage that they can be constructed easily. The implications of our findings for teaching Bayesian reasoning will be discussed.